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Abstract 

This comparative study paper seeks to investigate the nature of reported measures of school success currently reported 
in Ontario with two other Canadian jurisdictions with similar school systems and student populations (Alberta and 
British Columbia). As education in Ontario, Alberta and British Columbia is, for the most part, a government enterprise, 
much of the debate concerning the success of the school system is carried out in the court of public opinion. Following 
a short discussion of both the history and governance of education in each of these jurisdictions, current demographic 
trends that may influence public information needs will be discussed in detail. The measures currently reported by the 
Ontario government (graduation rates, and literacy and numeracy scores) will be compared with those employed by the 
other jurisdictions studied, as well as data historically reported in past Ministry of Education annual reports to 
determine which offers their populace a complete picture of the state of the school system. Data was gathered using a 
document analysis and a review of the reported measures as displayed on the ministry/department of education website 
for each jurisdiction studied. Both the utility of standardized testing as a policy tool and the contentious nature of these 
measures will be discussed in detail, including the problematic social justice and data manipulation issues clouding their 
use in Ontario and other jurisdictions in Canada and abroad. 

Keywords: education performance indicators, education policy, educational leadership, education system measures 

1. Introduction 

Few Canadian public services are subject to the same level of intense scrutiny, analysis and evaluation as the various 
provincial and territorial education systems. With education being, for the most part, a government enterprise, much of 
the debate surrounding the effectiveness of these education systems takes place in the court of public opinion. 
Hargreaves and Shelley (2009) note that this public scrutiny has only increased in the wake of the accountability 
movement was ushered into the education sector (and the whole of the public sector) in the mid-late 1980s. 

With the state of public education and test scores continually making headlines in major media outlets with national 
exposure (Boesveld, 2012; O’Toole, 2011; Simpson, 2011), it is surprising to find a dearth of literature investigating the 
nature, completeness and appropriateness of the data released by education authorities for public accountability 
purposes. This paper will place a focus on reported data released by the following three Canadian provinces: Alberta; 
British Columbia; and Ontario. The main method used by each of these jurisdictions (and many others worldwide) to 
demonstrate accountability to their populace, as well as report on and demonstrate the success of their respective 
education systems is through the reporting of student achievement outcomes data in relation to a set of agreed-upon 
academic standards (Knighton, Brochu & Gluszynski, 2010; Nichols & Berliner, 2007; Linn, 2000; Froese-Germain, 
1999). As the school districts and the departments or ministries of education in these jurisdictions surely are collecting a 
great deal more information than what was mentioned above, it begs an investigation of the types of data that are 
reported-, the appropriateness of those measures and how that data is utilized once it becomes part of the public sphere. 

This paper is divided into four sections. After a short overview of the jurisdictions, the first section of this paper will 
conclude with a discussion of some of the major issues involved with the public reporting of educational data, framed 
by the three problem-areas Fullan (1982) identified with large-scale provincial curricular assessments. The second 
section describes the methodology and the limitations of this study. The findings, grouped by the types of reported data 
released by the jurisdictions included in the study will comprise the third section of the study. The final section offers a 
short discussion of the implications that these findings have for current policy and future research, and some final 
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thoughts which conclude the paper. 

2. A Brief Overview of the Jurisdictions 

Snapshots of each of the three jurisdictions investigated in this study are presented below. 

2.1 Alberta 

According to Alberta Education (Alberta Education - Putting Students First, 2013), the province enrolled approximately 
425,000 students in the public school system in the 2012-13 academic year. $500 million of the provincial government’s 
$6.1 billion investment in education is earmarked for recent political commitments to build 50 new schools and 
renovate 70 more facilities. Alberta Education collects student outcomes data using criterion-referenced Provincial 
Achievement Tests (PAT). Students in grades 3, 6 and 9 are tested on a number of subjects, including language arts, 
French, science, social studies and mathematics. Though Alberta is considering alternatives to their current testing 
practices (Rodrigues, 2012), it currently tests students in more subjects and more often than the other jurisdictions 
studied. 

2.2 British Columbia 

British Columbia’s -funded school system served just fewer than 650,000 students during the 2011-12 academic year 
(K-12 Funding Allocation System, 2013). Total expenditures in the education portfolio in the same year was $4.7 billion 
(K-12 Funding Allocation System, 2013). Of the jurisdictions studied, British Columbia is the second largest, in terms 
of student population, but has the lowest overall expenditures. 

British Columbia conducts student achievement testing in a number of secondary school subjects in Grade 10 and up 
(BC Ministry of Education - Assessment, 2013; Scholarship funds with reach more students, support families, 2011), 
with success on these tests tied to post-secondary scholarship funding . In the elementary panel, criterion-referenced 
foundational skills assessments (FSA) are distributed to students in Grades 4 and 7 and tests student skill in numeracy, 
writing and reading comprehension (BC Ministry of Education - Assessment, 2013). British Columbia discloses the 
greatest amount of reported data in the jurisdictions studied. 

2.3 Ontario 

Comprised of 72 school districts that served in excess of 1,800,000 students in the 2010-11 academic year, Ontario’s 
-funded school system is both the largest and most populous in Canada (Education Facts, 2012). In the same year 
(Education Facts, 2012), the provincial government’s total investment in the education system totalled more than $20 
billion. Data related to process indicators, like those mentioned above, are reported on the Ontario Ministry of 
Education’s website (Education Facts, 2012), while student outcomes data is the responsibility of a government-agency 
called the EQAO (Education Quality and Accountability Office). 

Legislated into creation in 1996 following its inclusion as a key recommendation of the Royal Commission on 
Learning’s (1994) For the Love of Learning report. The EQAO is charged with collecting and analyzing data on 
Ontario’s -funded school system. Prior to the formation of the EQAO there were, “no such external, public checks on 
Ontario’s funded elementary school system were in place” (Johnson, 2005, p. 213). The EQAO collects a wide range of 
data, including process indicators, outputs and contextual factors that influence classroom learning as well as student 
achievement outcomes (EQAO: Education Quality Indicators, 2012). It is clear from looking at the provincial 
educational quality framework, the Ontario Ministry of Education reports only a sliver of the data they collect for 
internal purposes. 

3. Research Questions 

This inquiry will be guided by the following research question: 

What are the similarities and differences in the nature and types of education data reported by Alberta, British Columbia 
and Ontario? 

Answering this question will ideally provide an opportunity to learn from the policies enacted in the jurisdictions 
studied and identify any holes or missing links in the current measures after comparing and contrasting the perceived 
strengths and weaknesses in the reported data available in each of the jurisdictions studied. 

4. Literature Review 

Before moving forward, it is important to discuss some of the issues related to the gathering, analysis and reporting of 
system-level education data for public consumption. Despite Hart (2012) emphasizing that “province-wide testing 
enjoys widespread public support” (p. 15) in Ontario, and that, “64% (of the respondents) agreed that EQAO testing 
helps keep the educational system accountable to parents and taxpayers” (p. 19), there are a number of problematic 
aspects to the collection, analysis and reporting of student achievement data beyond simply demonstrating public 
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support for the practice. Fullan (1982) identified the following three problem areas associated with provincial curricular 
assessments, like those developed and delivered in the three jurisdictions studied: how to collect the data; how the 
information will be utilized and what information should be collected. The next section of this paper places an emphasis 
on discussing some of the issues and concerns associated with those problem areas. 

4.1 Issues Surrounding Data Collection 

As mentioned earlier, large scale assessments are the main mechanism used in Ontario, Alberta and British Columbia to 
gather much of the student achievement and outcomes data reported for accountability purposes. While a policy norm in 
education systems around the world (Cowley & Easton, 2013a; 2013b; Cowley, Easton & Thomas, 2011; Zwaagstra, 
2011), these large scale assessments, like the use of standardized tests in other jurisdictions (Geisinger, 2005; Kohn, 
2000), do not come without controversy. Much of this critique is related to perceived social and fiscal costs of 
conducting the assessments. Lotherington (2003) exemplifies much of the criticism regarding the “social'’ costs of the 
assessments by arguing that the tests are English-language dependent, paper-based, and culturally and historically 
Anglo-centric. Other critics (EQAO Testing, 2012), perceive the assessments to be a misuse of public funds that could 
be better spent elsewhere. Each of these concerns is discussed below. 

4.2 Social Justice / Equity Concerns 

Large scale testing has become a social justice issue in many jurisdictions due to allegations of bias emerging in terms 
of the following factors, among others: gender; ethnicity; second language learners; and socio-economic status/class 
(Geisinger, 2005; Kohn, 2000; McNeil, 2000, Froese-Germain, 1999). These concerns are particularly troubling and beg 
further investigation. While female students were producing better results than their male peers on EQAO assessments 
(2011), one has to wonder if a gender bias in the test itself was truly at play, or if the tests are simply reflecting an 
inequity in the school system as a whole. For instance, earlier Klinger, Shula and Wade-Woolley (2009) used focus 
groups to determine if the male to female achievement gap found in the EQAO data was consistent with general school 
performance. Their qualitative findings showed that female students, relative to their male peers, “have a significant and 
consistent advantage in literacy from an early age over boys” (p. 38). Since the test results seem to mirror classroom 
performance, perhaps this gender-based achievement gap found in the EQAO results is the result of numerous others 
factors, rather than the product of biased tests. Despite the EQAO assessments seeming to lack test-specific gender bias, 
this may not be the case when it comes to the other allegations of bias faced by other tests (Geisinger, 2005; Kohn, 2000; 
McNeil, 2000, Froese-Germain, 1999). It may be interesting to investigate that path of inquiry further in the future. In 
the meantime, it is vital that both the EQAO (EQAO, 2011; 2006; 2000), Alberta Education, the British Columbia 
Ministry of Education, and external stakeholders (e.g.. People for Education, 2011) continue to ensure that all 
assessment tools used for public accountability purposes are as fair and equitable as possible. 

4.3 Costs of Collecting the Reported Data 

A common argument by those in Ontario who view the EQAO assessments with a jaundiced eye (EQAO Testing, 2012), 
is that a number of other measures could be used to assess systemic performance without the expenditures tiedto large 
scale assessments. This issue seems to have been seriously distorted by special interest groups throughout the province. 
For example, the EQAO’s 2009-10 annual report reported approximately $33 million dollars in expenditures (EQAO, 
2011). However, the Elementary Teacher’s Federation of Ontario (ETFO), a vehement opponent of the EQAO’s 
large-scale, standards-based testing, lump the provincial Literacy and Numeracy Secretariat’s annual budget into the 
expenditures already reported by the EQAO to increase the total annual expenses of, what they call, “Ontario’s testing 
agenda” to upwards of $100 million dollars (EQAO Testing, 2012). This is a problematic and rather bizarre argument. 
The Literacy and Numeracy Secretariat was designed with a mandate to help raise student achievement on the whole 
(The Literacy and Numeracy Secretariat, 2012), and is not simply and organization tasked with solely “designing and 
mandating programs designed to raise test scores” (EQAO Testing, 2012), as the ETFO website suggests. It seems a 
rather large leap of faith would be necessary to “buy into” this line of reasoning. 

Considering that the government’s total investment in education was projected to be approximately $21 billion for the 
2011-12 academic year (Education Facts, 2012), the $33 million earmarked for the EQAO (EQAO, 2011) seems 
miniscule in comparison. It also seems an appropriate amount of funding to carry out their mandate. In the same vein, 
Alberta Education spends $12 million annually on their PATs in (Alberta Education - Putting Students First, 2013), a 
similar fraction of their annual budget, which approached $6 billion in the same year. While this data is unavailable for 
British Columbia, it seems continued debate surrounding the fiscal cost of the assessments has the potential to further 
the narrowness found in the measures currently being employed. This seems an unhealthy path to chart, especially 
considering the relatively small amount of capital necessary to conduct, analyze and report on the assessments. 
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4.4 What Information Should Be Collected? 

As is further discussed in the findings, there are numerous types of data that can be collected and reported back to the 
general public. However, all data is not of equal use for accountability and policymaking purposes. The two main types 
of data used for this purpose are process and outcomes data. While there seems to be a great deal of controversy 
surrounding student outcome measures acting as reported indicators or proxies of school success (Nichols & Berliner, 
2007; Stone, 1996; 1988, Fullan, 1982), those who aim to rely on process data for public accountability of policy 
formation also face a number of questions. 

Scheerens (1994) clearly demonstrates that current knowledge of school learning is too limited to rely on process 
indicators for accountability purposes. For instance, he notes that, “increasing the school day by half an hour leads to an 
expected gain in arithmetic progress by 3/10 of a month" (Scheerens, 1994, p. 28), is the type of detailed information 
needed in order to gain clarity around the utility of process information in an education system. The types and nature of 
data reported in all jurisdictions studied did not change in the 1970s, when the Ministry of Education made the decision 
to supplement these reports and release data related to process indicators in digest-sized volumes titled. Education 
Statistics (Ontario Ministry of Education, 1986; Ontario Ministry of Education, 1977) into the late 1980s. Like many 
other government functions, this process data is available on the websites for each of the ministries or departments of 
education included in this study (Alberta Education - Putting Student First, 2013; BC Ministry of Education - 
Assessment, 2013; Education Facts, 2012). 

There is also debate concerning the completeness and appropriateness of the current measures used in the jurisdictions 
studied. There are fundamental concerns with the use of quantitative performance measures to assess performance in the 
public sector, including education (Nichols & Berliner, 2007; Stone, 1988; Campbell, 1976). Though Stone (1988) 
mentions that, "the dominance of numbers as a mode of describing society and public policy in public policy 
discussions is only a recent, and perhaps temporary, phenomenon in cultural history" (p. 146), the utility and 
appropriateness of quantitative performance measures in education is a debate that has been raging since at least the 
middle of the past Century. 

As far back as 1956, Blau postulated that establishing performance standards could prompt public agencies (i.e., school 
districts and education systems) to lose sight of policy outcomes in an effort to achieve the highest performance ratings 
possible. In essence, Blau (1956) seems to be arguing that, whatever is being measured, rather than the process or goal 
being represented by the indicator, becomes important to people. This concept seems to be a precursor to Campbell’s 
law. After studying many public policy measures from jurisdictions around the world, Campbell (1976) established that, 
"the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption 
pressures and the more apt it will be to distort and corrupt the social processes it is designed to monitor” (p. 49). In 
terms of educational assessments, Campbell (1976) mentions that, “achievement tests, are in fact, highly corruptible 
indicators” (p. 53), and that test scores are no longer important as indicators when higher scores, rather than increased 
learning, becomes the goal of instruction. Nichols and Berliner (2007) also make mention of Campbell’s law and argue 
that, "when so much rides on a few solitary data points, then Campbell’s law predicts that humans become human, not 
surprisingly” (p. 79). The whole concept of people trying to "game” performance indicators in an effort to inflate their 
performance is particularly interesting, yet is only one of the issues related to the use of quantitative outcomes measures 
in education. 

While Stone (1988) agrees that, "people change the activities that are being measured” (p. 146), she also argues that the 
issues with quantitative performance measures begin before one even begins measuring the phenomenon in question. 
Before measuring anything, one must decide how it will be measured. As Stone (1988) notes; 

"there are many possible measures for any phenomenon and the choice among them depends on the purpose for 
measuring. The fundamental issues of any policy conflict are always contained in the question of how to count the 
problem” (p. 127). 

It is this question, concerning the completeness and appropriateness of the measures being employed, that appears to be 
the crux of the debate surrounding the public reporting of data in Ontario. 

4.5 How Is the Data Utilized? 

While the ministries and departments of education in the jurisdictions studied surely have internal uses for the various 
types of data they collect, this is not the focus of this section. Rather, it seems that Fullan (1982) was referring to the use 
(or misuse) of reported student outcomes data once it enters the public sphere. As such, this phenomenon will be focus 
of the following section. 

A key criticism of reporting student achievement data is that anybody or any institution has the opportunity to carve 
their own narrative out of the data once it becomes part of the public domain (Nicols & Berliner, 2007; Johnson, 2005). 
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The Fraser Institute (Cowley & Easton, 2013a; Cowley & Easton, 2013b; Cowley, Easton & Thomas, 2011), a 
pan-Canadian think-tank with “small-c” conservative ties, annually release their own analysis and interpretations of 
recent large scale tests scores in all three jurisdictions studied. The C.D. Howe Institute, another similarly-minded 
think-tank released a similar report in 2005. While the analysis driving the C.D. Howe Institute’s report (Johnson 2005) 
at least attempted to also determine key success factors associated with high scores on EQAO assessments, recent 
Fraser Institute report cards in each of the jurisdictions (Cowley & Easton, 2013a; Cowley & Easton, 2013b; Cowley, 
Easton & Thomas, 2011) seem to purport a punitive discourse of school reform rooted in the public shaming of schools 
that do not satisfy their criteria for success. 

This is particularly troubling for a number of reasons. In his seminal. The Meaning of Educational Change (1982), 
Fullan argues that, “crude public (especially media) presentations of school-by-school test and examination results in 
league tables...are damaging and humiliating to the worst performing schools” (p. 74). Compounding this problem is 
that think-tanks often combine a perceived level of credibility with the production of highly professional and accessible 
knowledge documents to sway public opinion. As Stone (1996) points out, most people are likely to be persuaded not 
by detailed discussions, but by, “condensed arguments that think-tanks portray through symbolism and their use of 
language”. On the other hand, Zwaagstra (2011) notes that a great deal of public information is misused by external 
research organizations and does not see that as a strong enough argument to stop either large-scale testing or the public 
reporting of student achievement data. 

5. Method 

As this study seeks to investigate and compare reported data released in three different Canadian jurisdictions, a 
document analysis seems the most appropriate method for carrying out this study. According to Gall, Gall and Borg 
(2005), a document analysis is a “qualitative investigation involving the study of written communications that are found 
in field settings” (p. 548). Creswell (2005) indicates that documents are useful to researchers as they contain 
“information from public sources about a sample or population” (p. 155). It made the most sense to gather the dataset 
used for this study from the jurisdictional department or ministry of education websites. 

5.1 Sampling 

A strategy called purposeful sampling was employed to gather the data included in this study. Gall et al. (2005) note that 
purposeful sampling is “the process of selecting cases that are likely to be ‘information rich’ with respect to the 
purposes of a particular (qualitative) study” (p. 554). Purposeful sampling was used in an effort to provide the most 
reasonable comparison possible. Alberta, British Columbia and Ontario were selected to be studied because they are 
Canada’s three most populous provinces that primarily use English as the main language of instruction in their schools. 
Because of these demographic similarities, these provinces could seemingly adapt reported measures used in the others 
jurisdictions studied while most likely dealing with little contextual baggage throughout the process. Another reason 
these three provinces were also selected for this comparison is because they have all instituted similar forms of 
criterion-referenced testing to obtain their reported student achievement data. Because of their similarities in population, 
policy and language of instruction, Alberta, British Columbia and Ontario are ideal for such a comparison. 

5.2 Limitations 

The main limitation facing this study is that data was collected from the education authority websites of the jurisdictions 
studied. Attempts were made to obtain the most recent data available for the jurisdictions under study at the time of 
writing. This meams the Alberta and British Columbia data is from 2013, and the Ontario data from 2012. However, it 
is worth mentioning that Alberta and British Columbia did report the same indicators in 2012 which were reported in 
2013, so the comparison is still valid. 

5.3 Delimitations 

Because it is only the reported indicators themselves that were analyzed, the motivations and processes used to develop 
the current measures in each of these jurisdictions is beyond the scope of this study. Though, perhaps conducting 
interviews with system leaders in each of these jurisdictions could be an interesting avenue for future research that 
could provide an answer to those questions. Another delimitation is that we simply do not possess the necessary 
knowledge about teaching and learning to simply view the process data and assume that the various inputs are being 
disseminated and working as planned. 

6. Findings 

An analysis of the reported education data available on the websites of each of the jurisdictions studied revealed that 
departments and ministries of education have a slew of available measures to report for accountability purposes. 
Collectively, Alberta, British Columbia and Ontario offer three different types of data for public consumption regarding 
their education systems. The findings are organized as a comparison of the following three types of reported data found 
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in each of the jurisdictions studied: process, outcomes, and satisfaction/engagement. Findings related to each type of 
data is discussed further in the following sub-sections. 

6.1 Process Data 

Each of the jurisdictions studied reports process data. Alberta, British Columbia and Ontario all report data related to 
student enrolment (including numbers surrounding special education), the number of teaching and administrative staff 
employed throughout their respective K-12 education systems. Alberta and Ontario also report the number of school and 
school districts in their respective jurisdictions. A comparison of the process data available in each of the jurisdictions 
can be found below in Table 1. Comparison of reported process data in each of the jurisdictions. 


Table 1. Comparison of reported process data in each of the jurisdictions 




Alberta 


British Columbia 


Ontario 

Process Data 

• 

# of schools and districts 

• 

Student enrolment (including the # 

• 

# of schools and school districts 


• 

Student enrolment 


with special needs) 


(boards and school authorities) 


• 

Special education data 

• 

#, average salary and years of 

• 

Special education data 


• 

Class size averages (by 


experience for all teachers and 

• 

# of teachers, administrators 



jurisdiction and subject) 


administrators 


and ECEs 




• 

Early learning development data 

• 

# of elementary and secondary 
schools 






• 

Total funding 

How is data 
collected 

• 

Administrative records 

• 

Administrative records 

• 

Administrative records 

Reported at 
what levels 

• 

District and provincial 

• 

District and provincial 

• 

Provincial 


There are number of differences and similarities in the process data reported by the three jurisdictions under study. All 
three jurisdictions report special education data. Alberta and British Columbia report process data related to student 
enrolment numbers, while Ontario does not. However, like Alberta, Ontario reports on the number of schools and 
school districts in the province. In terms of differences, Alberta makes class size averages readily available for public 
consumption, while British Columbia does the same for early learning development data. Both the former and latter are 
not reported in the other jurisdictions under study. Data for each of the jurisdictions was accessed using administrative 
records available online. All three jurisdictions report process data at the provincial level, while Alberta and British 
Columbia disaggregate process data at the district-level. One of the more interesting findings related to the process data 
is that Alberta reports their average class sizes aggregated by jurisdiction and subject. Considering that Ontario and 
British Columbia have instituted and publicized “caps” on class sizes in at least one division or panel, it is not surprising 
that they refrain from making these figures public, as all classes must remain at a certain level of enrolment. British 
Columbia also reports results related to their early learning development indicator tool (EDI), which measures both the 
school-readiness and socio-economic status for every region of the province. 

6.2 Outcomes Data 

The section describing the reported process datasets available in each jurisdiction was particularly short by design. It is 
the outcomes that really matter to people and each of the jurisdictions report similar outcomes measures. All 
jurisdictions studied report secondary school graduation or completion rates. Alberta and Ontario report five year 
completion rates, while British Columbia offers the same measure for public consumption, but with a six year outlook. 

Similarly, while all jurisdictions studied report student outcomes data, there is variation in the ways in which they go 
about collecting that data. Ontario conducts four different criterion-referenced tests, with two in each of the elementary 
and secondary panels. British Columbia takes a similar approach to reporting student outcomes in the elementary panel, 
but differs from Ontario in that they test student knowledge and comprehension in a number of secondary school 
subjects for students in grades 10 and up. As mentioned earlier, Alberta tests and reports more data than the other two 
jurisdictions studied. Both elementary and secondary students are tested in a variety of subjects. All jurisdictions studied 
report the aggregate results of these tests at the school, district and provincial levels. 
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Table 2. Comparison of reported outcomes data in each of the jurisdictions 



Alberta 


British Columbia 


Ontario 

Student Achievement 

• 

Provincial achievement 

• 

Foundation skill assessment 

• 

Assessment of reading. 

Outcomes - Test Data 


tests (grades 3, 6 and 9 - 


scores (grades 4 and 7 - 


writing and mathematics 



in various subjects) 


literacy, numeracy and reading 


(primary division) 


• 

Diploma examination 


comprehension) 

• 

Assessment of reading. 



results (Grade 12 students 

• 

Provincial required 


writing and mathematics 



- in various subjects). 


examinations (a number of 


(junior division) 





subjects for students in grade 

• 

Grade 9 assessment of 





10 and up). 

• 

mathematics 

Ontario secondary school 
literacy test. 

Student Achievement 

• 

Secondary school 

• 

Secondary school graduation 

• 

Secondary school 

Outcomes - Graduation 


completion rate (five 


rates (six year). 


graduation rates (five 

Data 


year). 

• 

Transition to post-secondary 
studies. 


year). 

Student Achievement 



• 

# and $ of examination 



Outcomes - Other Data 



• 

scholarships and awards. 

Grade to grade transitions. 



Participation in 

Alberta reports student 



Ontario reports student 

National/International 

achievement results achieved 



achievement results achieved 

Testing 

in 

the following studies: 



in 

the following studies: 


• 

School Achievement 



• 

Trends in Mathematics 



Indicators Program 




and Science Study 



(SAIP) 




(TIMSS) 


• 

Trends in Mathematics 



• 

Progress in International 



and Science Study 




Reading Literacy Study 



(TIMSS) 




(PIRLS) 


• 

Programme for 



• 

Programme for 



International Student 




International Student 



Assessment (PISA). 



• 

Assessment (PISA) 

Pan Canadian Assessment 
Program (PCAP) 

How is data collected 

• 

Criterion-referenced 

• 

Criterion-referenced testing 

• 

Criterion-referenced 



testing 

• 

Administrative records 


testing 


• 

Administrative records 



• 

Administrative records 

Reported at what levels - 

• 

Test data is reported at the 

• 

Test data is reported at the 

• 

Test data is reported at the 

test data 


school, district and 


school, district and provincial 


school, district and 



provincial levels. 


levels. 


provincial levels 

Reported at what levels - 

• 

Reported at the provincial 

• 

Transition/graduation data is 

• 

Reported at the provincial 

graduation data 


level. 


reported at the provincial 
level. 


level. 

Shown above, 


Table 2. Comparison of reported outcomes data in each of the jurisdictions displays a comparison of the reported 
outcomes data offered by Alberta, Ontario and British Columbia, respectively. A key difference in the types of outcomes 
data made available in each jurisdiction is that both Alberta and Ontario present student/systemic results in national and 
international tests and studies. For instance, both jurisdictions report their results on PISA (Programme for International 
Student Assessment), which is conducted by the OECD (Organization for Economic Development) and considered one 
of the more significant international comparative studies that collects and reports outcomes data from jurisdictions 
across the globe. 

6.3 Satisfaction/Engagement Data 

The third type of data reported in the jurisdictions studied is related to participant satisfaction and engagement in/with 
their provincial ministry of department of education. Alberta and Ontario have refrained from reporting any data related 
to these types of indicators (despite the Ontario School Climate Surveys seemingly collecting this data), whereas British 
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Columbia reports satisfaction and engagement data for both students and parents. British Columbia’s 
satisfaction/engagement data is collected using a survey tool and aggregated at the elementary and secondary levels. 

7. Discussion 

Conducting this comparison of the types and nature of reported education data released by Alberta, British Columbia 
and Ontario has presented a number of interesting avenues to explore for ministries and departments of education in 
these jurisdictions to present a more complete picture of systemic health and performance for accountability purposes. 
As all of my experience lies within the Ontario context, much of the following section will focus on how Ontario can 
offer stakeholders a more complete picture of the current state of -funded education in the province. 

Both Alberta and British Columbia conduct more testing than Ontario, with secondary school students participating in 
subject-specific testing. While I understand the utility standardized or criterion-referenced tests can have for 
policymaking, simply testing more subjects in Ontario is not a tack I would propagate to offer a more complete picture 
of systemic health and performance. Not only are there are a number of social justice and equity concerns which make 
testing more subjects increasingly problematic, but as mentioned earlier, the rollout, delivery and analysis of such tests 
have been perceived as a costly endeavor in Ontario and similar jurisdictions worldwide. 

Considering that each of the jurisdictions use and report similar indicators as proxies for student achievement, it begs 
the question of who decided or determined that these were the ideal reportable measures of school quality, or schooling 
in general. This swath of essentially agreed-upon measures is being used and reported by many jurisdictions worldwide 
seems to problematic, as all of the jurisdictions in this study fail to report any outcomes measures that are not tied 
directly to student achievement. As things like safe schools and character education increasing become priorities areas 
within public schools, some segment of the general public, composed of disparate groups with overlapping and 
competing interests, will call for some kind of accountability related to these concerns. 

Not only is the provenance of the current measures an intriguing avenue for future study, but the motives and 
explanations behind the measures currently in place seems to be a mystery. The current measures can easily be subject 
to manipulation by simply changing the assessment tools, or analyzing the data in different ways. With a multitude of 
available measures that are seemingly not subject to the same flaws, one wonders if the current reported outcomes 
measures have been selected because of they have these challenges. 

The other main difference in the types of reported data disseminated by these three jurisdictions is that British Columbia 
reports data related to student and parent satisfaction/engagement. While an interesting and encouraging tack, the 
reported lack of response rate to the parent/guardian surveys in the elementary (17% of potential respondents) and 
secondary (11% of potential respondents) panels is cause for concern. If the jurisdictions under study are able to 
generate an acceptable response rate on a survey to measure parent and student engagement or satisfaction, it would be 
a useful and meaningful addition to the jurisdictions current cache of reported measures. 

8. Final Thoughts 

As long as school-based assessments are unable to provide the system-level outcomes data vital for both accountability 
purposes and interpreting policy and programmatic needs, some form of large scale student assessment seems likely. 
This is despite the problems inherent in the use of quantitative indicators for decision and policymaking purposes, the 
distortion of the data by dubious independent organizations seemingly attempting to further their own interests and the 
multitude of issues surrounding system-level outcomes data, including some perceived social justice and equity 
concerns discussed earlier. 

Given the multitude of real and perceived issues inherent in the gathering and reporting of educational data for public 
consumption, one wonders if examining the indicators and measures used in other public service sectors, like health 
care, or even the corporate arena may provide valuable insight in either refining the current education data being data 
reported in Ontario or by providing reasonable and meaningful alternatives. 

In addition to analyzing the reported indicators found in similar jurisdictions and other sectors, questions like those 
which follow seem both relevant and appropriate for further study: 

• Who decides what data gets reported? 

• What is the provenance of the current measures? Are they still appropriate? 

• What processes (if any) are in place to obtain public input into what data is reported? 

• How are these measures more or less representative of systemic health (or better for policymaking and 
programmatic purposes) than the slew of others available?; and 

• Do current measures tell the complete story? Is that even possible? 
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While it was initially somewhat disheartening to feel like there are now have more questions, it is encouraging to have 
so many potential avenues for future inquiry. Perhaps conducting interviews with key informants involved in recent 
reform efforts in each of the jurisdictions studied, combined with the jurisdictional scans mentioned above could 
provide some answer. No matter what indicators are used to demonstrate the performance of whole education systems, 
it appears that they must speak to the needs and, if possible, the desires of all stakeholders. 
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